1 link tagged with all of: ai agents + web crawling + robots.txt + llms.txt
Click any tag below to further narrow down your results
Links
AgentReady is an open specification laying out what a website should do so AI agents can find, read, and act on it. The research behind it, drawn from over 250,000 agent runs, suggests that sites agents can't read get answered from other people's pages instead.
- Sites agents couldn't read were 3.7 times as likely to produce answers missing the facts the user asked for, and agent-ready businesses were clearly recommended about 1.9 times as often.
- In the traced runs, 82% of final answers traced back to a page the agent fetched, and docs pages were the most common source at 47%.
- Blocking is the big failure: not-agent-ready sites blocked agents 2.1 times as often, and a WAF returning a 403 or challenge page stops agents regardless of robots.txt.